Papers with SPARROW score
The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing instruction tuned large language models (LLMs) struggle to understand cross-lingual sociopragmatic meaning (SM) lack of comprehensive investigation into their ability to understand SM is partly due to SM not being adequately represented in any of the existing benchmarks. |
| Approach: | They evaluate the performance of instruction tuned large language models (LLMs) on a multilingual benchmark specifically designed for SM understanding. |
| Outcome: | The proposed benchmark outperforms instruction tuned large language models on a wide range of tasks but falls behind task-specific finetuned models. |